Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: parallelize Examples tests with turbo #4929

Closed
wants to merge 1 commit into from
Closed

ci: parallelize Examples tests with turbo #4929

wants to merge 1 commit into from

Conversation

mehulkar
Copy link
Contributor

@mehulkar mehulkar commented May 13, 2023 •
edited
Loading

This is an alternative approach to #4190. There is more boilerplate, but it models each test of an example as its own workspace, so turbo can parallelize its execution. If when/we turn on remote caching in CI that will help even more.

Reviewer notes (please read 😄 )

  • Even though we have turbo.json configuration for what contributes to the cache key, we are still using the Github Actions to limit when this runs to changes in examples/ and examples-tests/**. Since these tests do not exercise a locally built turbo, this is fine. In the future we may want to run examples tests against CLI changes as well, but we will cross that bridge when we get there.
  • Because the examples/ directory itself is a single workspace, any change to any example will cause a cache miss on all examples-tests. This is not ideal, but I'm not attempting to solve that here.
  • There are some funky tests that don't really make sense to me (e.g. npm with the with-yarn). I'm not planning to change/delete those in this PR, unless I've translated the matrix wrong.
  • We are now validating the output of turbo. I had to do this mostly for debugging issues with this PR, but ended up keeping it. The disadvantage is that if a new version of turbo is published and changes this output, we won't know until the next time these tests are triggered. This exacerbates an existing problem, so I'm noting here, but I think it's actually a good problem to surface when it happens. We could consider running these examples tests on publishing a new version.

@vercel
Copy link

vercel bot commented May 13, 2023 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
examples-designsystem-docs 🔄 Building (Inspect) Visit Preview May 17, 2023 9:40pm
turbo-site ✅ Ready (Inspect) Visit Preview 💬 Add feedback May 17, 2023 9:40pm
9 Ignored Deployments
Name Status Preview Comments Updated (UTC)
examples-basic-web ⬜️ Ignored (Inspect) Visit Preview May 17, 2023 9:40pm
examples-cra-web ⬜️ Ignored (Inspect) Visit Preview May 17, 2023 9:40pm
examples-gatsby-web ⬜️ Ignored (Inspect) Visit Preview May 17, 2023 9:40pm
examples-kitchensink-blog ⬜️ Ignored (Inspect) Visit Preview May 17, 2023 9:40pm
examples-native-web ⬜️ Ignored (Inspect) Visit Preview May 17, 2023 9:40pm
examples-nonmonorepo ⬜️ Ignored (Inspect) Visit Preview May 17, 2023 9:40pm
examples-svelte-web ⬜️ Ignored (Inspect) Visit Preview May 17, 2023 9:40pm
examples-tailwind-web ⬜️ Ignored (Inspect) Visit Preview May 17, 2023 9:40pm
examples-vite-web ⬜️ Ignored (Inspect) Visit Preview May 17, 2023 9:40pm

@github-actions
Copy link
Contributor

✅ This change can build next-swc

@mehulkar mehulkar changed the title Mk/examples parallel via turbo Parallelize Examples tests with turbo May 13, 2023
@github-actions
Copy link
Contributor

github-actions bot commented May 13, 2023 •
edited
Loading

⚠️ CI failed ⚠️

The following steps have failed in CI:

  • Turbopack Rust tests (mac/win, non-blocking)

See workflow summary for details

@github-actions

This comment was marked as outdated.

Sign in to view
@mehulkar mehulkar force-pushed the mk/examples-parallel-via-turbo branch from 178e5e7 to 676e97c Compare May 16, 2023 07:43
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as outdated.

Sign in to view
@mehulkar mehulkar force-pushed the mk/examples-parallel-via-turbo branch from 676e97c to 6b6b4b1 Compare May 16, 2023 22:54
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

This comment was marked as off-topic.

Sign in to view
@mehulkar mehulkar force-pushed the mehulkar/turbo-1108-remove-turborepo-deps-for-examples-tests branch from 874378c to cb06a3e Compare May 17, 2023 16:22
@mehulkar mehulkar force-pushed the mk/examples-parallel-via-turbo branch from 5fb11bc to 4d35cd1 Compare May 17, 2023 16:22
@turbo-orchestrator turbo-orchestrator bot added the area: examples Improvements or additions to examples label May 17, 2023
@mehulkar mehulkar mentioned this pull request May 17, 2023
Merged
@mehulkar mehulkar force-pushed the mk/examples-parallel-via-turbo branch from dfa67d8 to eb5420f Compare May 17, 2023 16:52
@turbo-orchestrator turbo-orchestrator bot removed the area: examples Improvements or additions to examples label May 17, 2023
@mehulkar mehulkar force-pushed the mk/examples-parallel-via-turbo branch from eb5420f to 2ecf9e0 Compare May 17, 2023 17:00
@mehulkar mehulkar force-pushed the mk/examples-parallel-via-turbo branch from 2ecf9e0 to b84f02d Compare May 17, 2023 17:07
@mehulkar mehulkar force-pushed the mehulkar/turbo-1108-remove-turborepo-deps-for-examples-tests branch from cb06a3e to 723aa4d Compare May 17, 2023 17:07
@mehulkar mehulkar force-pushed the mk/examples-parallel-via-turbo branch from b84f02d to 541de93 Compare May 17, 2023 17:08
@github-actions
Copy link
Contributor

Linux Benchmark for 6a938e5

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 8347.88µs ± 83.47µs 8380.78µs ± 82.20µs +0.39%
bench_hmr_to_eval/Turbopack CSR/1000 modules 7227.06µs ± 34.14µs 7306.05µs ± 81.15µs +1.09%
bench_startup/Turbopack CSR/1000 modules 888.86ms ± 1.64ms 891.49ms ± 1.85ms +0.30%

@mehulkar mehulkar force-pushed the mehulkar/turbo-1108-remove-turborepo-deps-for-examples-tests branch from 723aa4d to c620266 Compare May 17, 2023 17:45
@mehulkar mehulkar changed the base branch from mehulkar/turbo-1108-remove-turborepo-deps-for-examples-tests to main May 17, 2023 17:54
@github-actions
Copy link
Contributor

Linux Benchmark for 8856f79

Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 8898.69µs ± 159.31µs 8330.21µs ± 38.50µs -6.39% -2.01%
bench_hmr_to_eval/Turbopack CSR/1000 modules 7175.65µs ± 46.58µs 7015.07µs ± 24.98µs -2.24% -0.25%
Click to view full benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 8898.69µs ± 159.31µs 8330.21µs ± 38.50µs -6.39% -2.01%
bench_hmr_to_eval/Turbopack CSR/1000 modules 7175.65µs ± 46.58µs 7015.07µs ± 24.98µs -2.24% -0.25%
bench_startup/Turbopack CSR/1000 modules 880.56ms ± 2.14ms 878.81ms ± 3.26ms -0.20%

@github-actions
Copy link
Contributor

MacOS Benchmark for 8856f79

Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 25.74ms ± 0.11ms 31.18ms ± 0.80ms +21.16% +14.02%
Click to view full benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 25.74ms ± 0.11ms 31.18ms ± 0.80ms +21.16% +14.02%
bench_hmr_to_eval/Turbopack CSR/1000 modules 25.83ms ± 0.64ms 25.54ms ± 0.41ms -1.15%
bench_startup/Turbopack CSR/1000 modules 2967.42ms ± 118.10ms 2749.59ms ± 201.78ms -7.34%

@mehulkar
ci: make every examples test a workspace
f9cec7d 
This setup will allow turbo to parallelize and cache execution of these
tests. It also reduces the number of Github Workflow runs that are
triggered when an example changes, greatly reducing the amount of
computing power required to run the tests.

This commit does not change the fact that all examples tests are run
when any of the examples change. We may be able to wire up dependencies
more carefully in the future, but this is not feasible at the moment
because each example itself is not a workspace.
@mehulkar mehulkar force-pushed the mk/examples-parallel-via-turbo branch from 397ee7a to f9cec7d Compare May 17, 2023 21:38
@github-actions
Copy link
Contributor

Linux Benchmark for 1100d6a

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 9136.75µs ± 50.41µs 9146.88µs ± 47.98µs +0.11%
bench_hmr_to_eval/Turbopack CSR/1000 modules 8155.78µs ± 51.11µs 8549.42µs ± 383.30µs +4.83%
bench_startup/Turbopack CSR/1000 modules 905.67ms ± 2.38ms 915.78ms ± 5.20ms +1.12%

@mehulkar
Copy link
Contributor Author

Closing in favor of #5011, because of so much noise from automation in this PR

@mehulkar mehulkar closed this May 17, 2023
@mehulkar mehulkar deleted the mk/examples-parallel-via-turbo branch May 17, 2023 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area: ci
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@mehulkar